Implementing Lighting Control of a User Environment

ABSTRACT

A system and method for controlling lighting conditions in a user environment whereby bandwidth consumption may be managed is disclosed. An inner frame area and an outer frame area adjust certain lighting conditions in the user environment in response to certain lighting conditions detected by the image capture device as those conditions affect bandwidth consumption during transmission of the image capture data. The frame areas may be dynamically controlled as to affect the brightness and/or color of the particular user environment.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation and claims the prioritybenefit of U.S. patent application Ser. No. 11/744,816 filed May 4, 2007and entitled “Bandwidth Management Through Lighting Control of UserEnvironment Via a Display Device,” which is a continuation-in-part andclaims the priority benefit of U.S. patent application Ser. No.11/624,886 filed Jan. 19, 2007 and entitled “Lighting Control of a UserEnvironment via a Display Device,” which claims the priority benefit ofU.S. provisional patent application No. 60/798,112 filed May 4, 2006 andentitled “Lighting Control of a User Environment via a Display Device.”The disclosure of each of the aforementioned applications isincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to the generation of visual datathrough an image capture device during an audio/visual session such asan ‘audio/visual chat’ session or during video game play. Morespecifically, the present invention relates to control of lightingconditions in a user environment whereby the bandwidth consumed bytransmission of image data generated and/or captured during suchaudio/visual sessions may be effectively managed.

2. Description of the Related Art

With the increased processing capabilities of various computing systems,new methods for interacting with those computer systems have becomeavailable. For example, a variety of input devices employing video imagecapture allow user control of or interaction with objects on a graphicaldisplay such as a video monitor.

Such video input devices often are responsive to the movement orposition of a user in the field of view of an image capture device.Video image processing translates the movement of the user that has beencaptured as a sequence of video images into signals that may be used,for example, to control a character or avatar in a video game.Alternatively, image processing may generate a video image to bedisplayed in a ‘chat’ session in a manner similar to a video conference.

An image capture device typically scans a field of view in which a usermay be present (e.g., a user environment such as an office, game room,living room or the like). The captured video image may be applied to avideo digitizer that provides digital output to a processor thatanalyzes and processes the digital information received from thedigitizer. Based upon the position or movement of the participant in thefield of view, the processor produces signals that are used by thegraphics generating system to move objects on the display. Similarly,the system may generate an image of the user for transmission to anotheruser in a chat session.

The output of the devices on graphical displays can be significantlyaffected by a variety of factors, especially lighting conditions in theuser environment. The computer processing time required for imageprocessing in an ideal user environment may be extensive and complex andtends to require substantial computing and/or temporal resources. A userenvironment that is overexposed or underexposed as to particular lightsources only complicates such processing activities in that the systemmust compensate for the adverse lighting conditions thereby resulting inslower computing operations and affecting any real-time interactions. Insome instances, the lighting conditions are so adverse that a computingsystem cannot compensate for the particular environment conditions andinaccurate data—if any—is generated thereby resulting in incorrect gamecontrol or the generation of poor chat session video.

Similarly, lighting conditions may contribute to the amount of bandwidthconsumed by transmission of a captured video image (or images). This maybe especially in the context of real-time, full-motion video as may begenerated in the context of a video chat session. Video signals, bytheir very nature, consume large amounts of bandwidth notwithstandingthe application of a variety of compression techniques (e.g., lossycompression) as are known in the art. An overwhelming presence of acertain color or colors in a particular environment (e.g., as might begenerated by bright lights of a particular color) may be captured by animage capture device. Transmission of that image and its excessive colorsaturation may increase an overall file size and lend to excessivebandwidth consumption. Excessive bandwidth consumption may, in turn,contribute to a degraded conferencing experience (e.g., jitter).

There is a need for an image capture device whereby the lightingconditions of various user environments can be automatically anddynamically controlled subject to the particular requirements of thecapture device or hardware and software related thereto. Additionally,there is a need for lessening the computational burdens of an imageprocessing system coupled to such a device whereby the system mayfunction substantially in real-time thus providing the user with anatural interaction experience with regard to a game, chat session, orany other interaction involving the aforementioned image capture device.Further still, there is a need in the art to reduce bandwidthconsumption that may occur as a result of the capture of certain imagedata that increases overall transmission file size.

SUMMARY OF THE INVENTION

In an embodiment, a system for managing bandwidth consumption throughcontrol of lighting conditions in a user environment is disclosed. Animage capture device captures image data in a user environment, which isthen transmitted over a communications network. A computing device maybe coupled to the communications network to measure consumption ofbandwidth related to transmission of the captured image data over thecommunications network. A display device may display an inner frame areaand an outer frame. The inner frame area and the outer frame areacontrol at least one lighting condition in the user environment inresponse to the consumption of bandwidth as measured by the computingdevice.

In another embodiment of the present invention, a method for managingbandwidth consumption through control of lighting conditions in a userenvironment is disclosed. Image data is captured and transmitted over acommunications network. Consumption of bandwidth related to transmissionof the captured image data over the communications network is measured.An inner frame area and an outer frame area may be generated on adisplay device in the user environment. At least one lighting conditionin the user environment may be controlled via the inner frame area andthe outer frame area in response to the consumption of bandwidth asmeasured by the computing device.

A video conference system includes a first image capture device and asecond image capture device. Each device is configured to capture userenvironment image data in a respective user environment and fortransmission over a communications network. A conference managementserver receives the first user environment image data and the seconduser environment image data and measures consumption of bandwidthrelated to transmission of the first user environment image data and thesecond user environment image data over the communications network. Afirst user client computing device and a second user client computingdevice are each associated with a respective image capture device. Thefirst and second user client computing devices receive lighting controlinstructions from the conference management server in response to theconsumption of bandwidth as measured by the conference managementserver. The lighting control instructions received from the conferencemanagement server include generating an outer frame area and an innerframe area on a display device at both the first user client computingdevice and second user client computing device, the outer frame area andinner frame area collectively controlling at least one lightingcondition in the respective user environment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary system for generating real-time,three-dimensional, interactive environment data according to anembodiment of the present invention.

FIG. 2 illustrates a block diagram of one embodiment of a clientcomputing device for generating real-time, three-dimensional,interactive environment data.

FIGS. 3A and 3B illustrate an exemplary video display device comprisingan inner frame area and an outer frame area for implementing lightingcontrol of a user environment in accordance with one embodiment of thepresent invention.

FIGS. 4A and 4B illustrate an exemplary video display device reflectingdifferent degrees of illumination intensity in the outer frame inaccordance with one embodiment of the present invention.

FIG. 5A illustrates an exemplary method for implementing lightingcontrol of a user environment through varying illumination intensity ina video display device in accordance with one embodiment of the presentinvention.

FIG. 5B illustrates an exemplary method for managing bandwidth withrespect to the generation and transmission of image capture data inaccordance with one embodiment of the present invention.

FIG. 6 illustrates an exemplary network of real-time, three-dimensional,interactive environment data generating systems as may be utilized in a‘chat’ session or interactive video game in accordance with oneembodiment of the present invention.

DETAILED DESCRIPTION

FIG. 1 illustrates an exemplary system 100 for generating real-time,three-dimensional, interactive environment data according to anembodiment of the present invention. The system 100 comprises an imagecapture device 110, a video display device 120, and a client computingdevice 130 having processor functionality such as a home entertainmentsystem for video games. The client computing device 130 may be furthercoupled to a controller device 140 as is shown in FIG. 1. Generally, auser and a related environment, such as a living room, are locatedwithin the field-of-view of the image capture device 110.

The client computing device 130 may be implemented as a PlayStation® 3from Sony Computer Entertainment Inc. It should be noted, however, thatthe client computing device 130 and its processor functionality may beimplemented in other types of computing devices such as personalcomputers, workstations, laptop computers, wireless computing devices,portable media devices, or any other type of computing device that maybe capable of receiving and processing graphical image data.

Video display device 120 may be any device configured for the display ofvideo data such as a television or computer monitor. In one embodiment,video display device 120 receives graphical image data from clientcomputing device 130 via an AV MULTI OUT connector that may be coupledto an integrated audio/video cable and the requisite video and audioconnectors. Various other connection mechanisms may also be used, forexample, S VIDEO, an RFU adaptor, component video input (Y CB/PB CR/PR),and HDTV input (Y PB PR) as well as the High Definition MultimediaInterface (HDMI) and Digital Visual Interface (DVI). In someembodiments, such as a portable media device, the client computingdevice 130 and video display device 120 may be integrated as is the casein a PlayStation® Portable from Sony Computer Entertainment Inc.

Image capture device 110 may be, in one embodiment of the presentinvention, a color digital camera device with functionality similar tothat of a webcam. Image capture device may be coupled to clientcomputing device 130 via a USB cable. In some embodiments, image capturedevice 110 may utilize a wireless transmission protocol to exchange datawith the client computing device 130 (e.g., 802.11x). Image capturedevice may also include commercially available dedicated videoconferencing products such as those manufactured by Polycom, Inc. ofPleasanton, Calif.

Image capture device 110 extracts specific information from capturedmulti-dimensional image data for subsequent presentation to a user ofthe device (e.g., a color image on visual display device 120 or fortransmission to another user over a network as in FIG. 6 below). Forexample, captured image data may be integrated into a video game orother user-centric application (e.g., a workout or training video).Captured image data may also be utilized in a video conference or‘audio/visual chat session’ wherein the captured data may be transmittedto another user's computing device for display.

Image capture device 110 may be capable of capturing and mapping inthree-dimensions in addition to normal two-dimensional video imagery.Similar to normal cameras, image capture device 110 capturestwo-dimensional data for a plurality of pixels that comprise the videoimage. These values are color values for each pixel—generally red,green, and blue (RGB) values. In this manner, objects captured by theimage capture device 110 appear as two-dimensional objects on a monitor.

Unlike a conventional camera, however, image capture device 110 may alsocapture depth values in a particular field-of-view (e.g., a particularuser environment). That is, image capture device 110 may capture the xand y components of an environment using RGB values for each pixel inthe environment in addition to a z-component, which represents a depthvalue for the environment. (i.e., the z-axis).

In operation, a z-value may be captured for each pixel of the scene,each z-value representing a distance from the camera to a particularobject in the scene corresponding to the related pixel. A maximumdetection range may be defined beyond which depth values will not bedetected. Through the use of z-value capture, each object can be trackedin three-dimensions whereby the z-values along with the two-dimensionalpixel data can be processed to create an enhanced three-dimensionalinteractive environment for the user.

An input image processor (not shown) at client computing device 130translates the captured video images and depth data into signals thatare delivered to an output image processor (not shown) at the clientcomputing device 130. The output image processor may be programmed toeffect movement and status of virtual objects on the video displaydevice 120 in response to signals received from the input imageprocessor. In some embodiments, input and output image processors may bean integrated part of the hardware and software configuration of theimage capture device 110. In other embodiments, such processingfunctionality may be partially distributed over a network like thatshown in FIG. 6, such processing taking place at, for example, aconferencing management server. Some embodiments of the image capturedevice 110 may also be configured for the capture and subsequentprocessing of visual data in high-definition.

Various image processing techniques allow a user to interact with theimage capture device 110 device using motion, color detection and, insome embodiments, sound through a built-in microphone or other audioinput device (not shown) coupled to the image capture device 110 orclient computer device 120. Certain interaction techniques are disclosedin U.S. patent publication number US 2004-0207597 A1 for a “Method andApparatus for Light Input Device” and U.S. Pat. No. 7,102,615 for a“Man-Machine Interface Using a Deformable Device.” A prop-input deviceis disclosed in U.S. patent publication number US 2004-0239670 A1 for a“System and Method for Providing a Real-Time Three-DimensionalInteractive Environment” as well as U.S. Pat. No. 6,795,068 for a“Method for Mapping an Object from a Two-Dimensional Camera Image to aThree-Dimensional Space for Controlling Action in a Game Program.” Thedisclosures of all of the aforementioned applications and patents areincorporated herein by reference.

In one embodiment of the present invention, the image capture device 110may be compact in design allowing for placement on top of a videoconsole or television. Image capture device 110 may comprise a pivotthereby allowing for positioning of the device and its relatedfield-of-view. The image capture device 110 may further comprise arotational ring around the camera lens for manual focus control. Someembodiments of the image capture device 110 may provide for automatedpositioning and focusing through the use of a directional-sensitivemicrophone that tracks a source of audio (e.g., an individual who isspeaking) and the focus of the device 110 may be subsequently adjustedaccording to the position of the audio source relative to the imagecapture device 110.

Referring now to FIG. 2, a block diagram of one embodiment of a clientcomputing device 200 (130) is illustrated. Client computing device 200may be used to aid in the generation of real-time, three-dimensional,interactive environment data. The client 200 may be communicativelycoupled to image capture device 110 in order to generate theaforementioned environment data.

The client 200 may comprise, but is not limited to, a main memory 202, acentral processing unit (CPU) 206, vector processing units VU0 204 andVU1 208, a graphics processing unit (GPU) 210, all of which may becoupled via a bus 236 to an input/output processor (IOP) 212. The client200 may also comprise an IOP memory 214, a controller interface 216, amemory card 218, a Universal Serial Bus (USB) interface 220, and an IEEE1394 interface 222. The client 200 may further include an operatingsystem read-only memory (OS ROM) 224, a sound processing unit (SPU) 226,an optical disc control unit 228, and a hard disc drive (HDD) 230, allof which may be connected via a bus 238 to IOP 212.

Some embodiments of the client 200 may also include a network adaptor240, which may offer an Ethernet connection 242 and/or telephonyconnection 244. The client 200, in one embodiment, may be an electronicgaming console although client 200 (or portions thereof) may also beimplemented as a general-purpose computer, a set-top box, a hand-heldgaming device, or in a mobile device such as a cellular phone. It shouldfurther be noted that various other system architectures may be utilizedwithin the scope of the present invention such as the computerarchitecture and high speed processing model disclosed in U.S. patentpublication number 2002-0138637 for a “Computer Architecture andSoftware Cells for Broadband Networks,” the disclosure of which isincorporated herein by reference.

The CPU 206, the VU0 204, the VU1 208, the GPU 210, and the IOP 212communicate via a system bus 236. The CPU 206 communicates with the mainmemory 202 via a dedicated bus 234. The VU1 208 and the GPU 210 may alsocommunicate with one another via a dedicated bus 232. The CPU 206executes programs stored in the OS ROM 224 and the main memory 202. Themain memory 202 may contain pre-stored programs and may also containprograms transferred via the IOP 212 from a CD-ROM, DVD-ROM, or otheroptical disc (not shown) using the optical disc control unit 228. TheIOP 212 controls data exchanges between the CPU 206, the VU0 204, theVU1 208, the GPU 210 and other devices of the system 200, such as thecontroller interface 216, or from other such systems via the networkadaptor 240.

The GPU 210 executes drawing instructions from the CPU 206 and the VU0204 to produce images for display on a display device (not shown). TheVU1 208 transforms objects from three-dimensional coordinates totwo-dimensional coordinates, and sends the two-dimensional coordinatesto the GPU 210. The SPU 226 executes instructions and processes data toproduce sound signals that are output on an audio device (not shown).

A user of the client 200 provides instructions to the CPU 206 via acontroller coupled to the controller interface 216. Controller may beany control device, for example, a joystick, a set of directionalbuttons, and/or other control buttons. An exemplary controller isillustrated in FIG. 1 (140). A user may instruct the CPU 206 to storecertain information on the memory card 218, which may be removable(e.g., a flash memory or other non-volatile memory card), or mayinstruct a character in a game to perform some specified action. Otherdevices may be connected to the system 200 via the USB interface 220 andthe IEEE 1394 interface 222. As previously noted, the image capturedevice 110 may be coupled to the client 200 utilizing a USB connectoror, in another example, a wireless Ethernet network through Ethernetconnection 242.

In that regard, some embodiments of the client 200 comprise a networkadaptor 240 that provides the hardware functionality necessary for theclient 200 to connect to a network. The network adaptor 240 maycomprise, for example, a system connector that operates to connect theadaptor 240 to the client computing device 200 through an expansion busconnector 246. The network adaptor 240 may also comprise a powerconnector and data connector to allow for the provisioning of power fromthe client 200 to the adaptor 240 and the exchange of data between theclient 200 and the adaptor 240. Network adaptor 240 may be fullyintegrated with the client 200 or may be a detachable hardware devicethat may be implemented in older legacy client devices 200.

In some embodiments of the present invention, the network adaptor 240may also require the installation of certain software on the client 200to allow for identification and connection to a particular IP addressand/or dial-up to a particular Internet Service Provider. Software mayalso provide other functionalities, such as the creation and maintenanceof user profiles, in addition to functional interaction between theclient 200 and the network adaptor 240. Such software or data related tosuch functionality may be embodied on CD-ROMs for games or applicationrequiring a network connection; stored on memory card 218; or part of afirmware upgrade.

The network adaptor 240 may also comprise an Ethernet connection 242.Through the Ethernet connection 242, a network cable (e.g., a 100Base-TX or 10-Base T) may be coupled to the network adaptor 240 forconnection to a network. The network cable may, for example, becommunicatively coupled to a DSL or cable modem. The network cable mayalso be communicatively coupled to, for example, a router via a LANport; the router may then be coupled to a DSL or cable modem through aWAN port. In further embodiments, the Ethernet connection 242 may allowfor a network cable to be connected to a wireless Ethernet bridge. Thewireless Ethernet bridge may be communicatively coupled to a wirelessrouter utilizing, for example, an 802.11x protocol. The wireless routermay be further communicatively coupled to a cable or DSL modem.

The network adaptor 240 may also comprise a telephony connection 244.Through the telephony connection 244, a standard telephone line with,for example, an RJ-11C telephone connector may be connected to thenetwork adaptor 240 and a telephone wall jack. In this regard, thenetwork adaptor 240 may further comprise modem functionality such thatthe client device 200 may communicate data over the public switchedtelephone network via the telephony connection 244.

FIG. 3A illustrates an exemplary video display device 300 comprising aninner frame area 310 and an outer frame area 320 for implementinglighting control of a user environment in accordance with one embodimentof the present invention. Lighting control of the user environment mayalso be used in the context of bandwidth management as further discussedherein. Video display device 300 corresponds to the video display device120 originally referenced in FIG. 1 and coupled to client computingdevice 130 (client 200 of FIG. 2) for the display of images or otherdrawing instructions by GPU 210. Video display device 300 may alsocomprise audio output functionality (e.g., a built-in speaker system) toemit audio signals produced by SPU 226.

Inner frame area 310 and outer frame area 320 are artificial boundariesthat may be created by client computing device 130 as the result ofdrawing instructions from the CPU 206 being executed by GPU 210. Thedimensions of these areas may be determined by or in light of aparticular software title being executed (e.g., a video game titleutilizing an image capture device 110) or by a server component as mightbe utilized in an audio/visual ‘chat’ session as is discussed in furtherdetail in FIG. 6. In some embodiments—for example, in the case of a 4:3game shown on a 16:9 display—the outer frame area 320 may be the extra‘dead space’ on the sides of a widescreen display.

Referring to FIG. 3B, the size of outer frame area 320 is larger thanthat of the corresponding frame area 320 in FIG. 3A. Accordingly, theinner frame area 310 of FIG. 3B is smaller than that of thecorresponding frame area 310 in FIG. 3A. The inner frame 310 and outerframe 320 may also be subject to particular user configurations orsettings that may be saved on memory card 218 or some other memorydevice coupled to the client computing device 130. Inner frame 310 andouter frame 320 may be adjusted in response to certain determinationsmade by client computing device 130 or in response to certaininstructions received from another computing device over a network.

In one embodiment of the present invention, video game data will bedisplayed in the inner frame area 310. This could include image data ofthe user as might be generated by the image capture device 110 or aderived visual game state based upon the image data of the user. Forexample, the image capture device 110 may capture an image of the userfor insertion into a game environment in the form of an avatar.Similarly, the image capture device 110 may generate data used todetermine whether the user came into contact with a particular portionof the game environment by ‘grabbing’ at an object or providing inputdata through a prop device. This real-world activity or interaction maybe reflected in the game environment as a result of the user, forexample, swinging at a baseball with a baseball bat prop device in aninteractive baseball game. An image capture device 110 tracks andcaptures the swinging of the baseball bat prop device and, through thenecessary images processors, translates that real-world activity intoon-screen display data.

The inner frame area 310 may also be used to display ‘chat’ session datasuch as the image of a remote participant communicating with the user.The inner frame area 310 may also display multi-media data generated bythe remote participant and transmitted to the present user in the formof a textual ‘instant message’ or a short video clip. Various types ofinteractive data may also be displayed in the inner frame area 310 suchas an interactive game (e.g., chess or checkers) or a collaborativedocument (e.g., a report that is being concurrently edited by one ormore parties).

As has been previously noted, image capture device 110 may requirecertain light levels or light conditions in order for the image capturedevice 110 to function properly or optimally. If the environment inwhich the image capture device 110 is operating in lacks sufficientlight, certain data (e.g., user interactions or prop input data) may notbe detected because the image capture device 110 cannot properly discernvarious x-y-z position values, color differentiations, or otherwisedetect motion of a user or object. Image capture device 110, inconjunction with software and/or hardware operating internal to theimage capture device 110, at the client computing device 130, and/or byan intermediate server as is discussed in FIG. 6, will determine whetherthere is proper light present in the environment to properly capturedata (e.g., with respect to a target histogram). Similar determinationsmay be made with regard to, for example, the presence of too much lightif the image capture device 110 or a computing device (130) coupled tothe image capture device 110 is configured with an infrared (IR)receiver to receive commands from a remote control (e.g., cameraon/camera off). Various other spectral emissions may be subject to suchuser environment light control.

In the context of managing bandwidth consumption as discussed furtherherein, determinations may be made with respect to the presence of toomuch light or over-saturation with respect to a particular color. Theseexcesses may result in spikes in bandwidth consumption when transmittingimage or video data captured and/or generated by image capture device110. Spikes in bandwidth consumption may result in a less than optimalconferencing experience, especially in constricted bandwidthenvironments.

Image capture device 110 in conjunction with the aforementioned softwareand/or hardware operating internal to the image capture device 110, atthe client computing device 130, and/or by an intermediate server asdiscussed in FIG. 6 may determine, for example, that too much red lightis present in the environment. This light may come from an externalsource such as natural sunlight through a window proximate the userenvironment, from an artificial lighting source in the user environment(e.g., an overhead lamp), or from the display device 300 itself.Utilizing the various display and frame adjustment techniques discussedherein, certain colors may be ‘muted,’ ‘countered’ or otherwise‘cancelled’ through additive (e.g., mixing of red, green, and blue) andsubtractive (e.g., subtracting a hue out of a color scheme by addingmore of another) color techniques as are known in color theory.

In the event that there is a determination that insufficient lightexists to properly capture image data or that too much light or aparticular hue of light exists and is adversely affecting bandwidthconsumption, the outer frame area 320 may be used to provide additionallight to the user environment such that the video display device 300operates as an artificial light source (e.g., certain RGB values ofindividual pixels in the display 300 are adjusted as to average anecessary user environment light level). That is, the outer frame area320 will become visibly brightened with a particular color and/orintensity of light such that the video display device 300 lights theenvironment proximate the image capture device 110 whereby image datamay be properly captured for subsequent processing.

For example, a user may be utilizing the image capture device 110 toplay the aforementioned simulated baseball game with a prop device. Assuch, the user may find an avatar of themselves inserted into the ‘homeplate’ video game environment and displayed in the inner frame area 310of FIG. 3A. If the image capture device 110 and related processingutilities determine that there is insufficient light in the sceneenvironment to properly track the prop device or any other aspect of theuser, the video game software operating at client computing device 130may (via CPU 206) cause the execution of rendering instructions by theGPU 210 that causes the rendering on the display device 300 of a white‘filler’ in the outer frame area 320 (i.e., the execution of drawinginstructions that causes the individual pixels within the outer framearea 320 to display a particular luminance and/or shading). The innerframe area 310 would continue to display the baseball video game andrelated information while simultaneously becoming surrounded by a white‘halo’ in the outer frame area 320.

This white ‘halo’ displayed by the video display device 300 will ‘spill’into the real-world environment of the user—that is, the light emittedby the display and measured in candelas per meter squared(cd/m²)—whereby the video display device 300 actually becomes analternative light source. This additional light generated by the videodisplay device 300 will result in the generation of sufficientadditional light to allow for the image capture device 110 to properlycapture image data for processing. The intensity of the ‘halo’ emittedby the outer frame area 320 can be controlled in part by the particularrendering instructions from GPU 210; that is, how white (what shade ofwhite) and how bright (the intensity thereof).

Similarly, if a particular environment exhibits too much of a particularhue of light such that it is degrading transmission of image data (i.e.,consuming excess bandwidth), the color and/or intensity of the ‘halo’ ofthe outer frame area 320 (and/or the inner frame area 310) may beadjusted and manipulated to compensate for the same. In the context ofthe Munsell color system, excess saturation (also referred to as chroma)of one particular hue (red) may be countered by a hue of the oppositecolor (blue/green). The particular saturation/hue combination may bewholly opposite, slightly opposite, or substantially opposite the excesssaturation depending on particular network conditions.

In some embodiments of the present invention, the particular hardwareconfiguration of the display device 300 may be taken into account suchthat the GPU 210 may optimize the resources at its disposal with regardto manipulating the outer frame area 320 and the light emitted by thesame. These configurations may be automatically detected by the clientcomputing device 200 or provided to the device 200 through user input asto particular configurations, the brand and model of the display device(which in turn corresponds to particular operating specifications), andso forth. In some embodiments, device 200 may consult a look-up table orother data store (either locally or over a network) to identifyparticular operating capabilities (e.g., bit depth) for a particulardisplay model if those capabilities are not immediately identified bythe user or through certain plug-and-play (PnP) functionality.

For example, a liquid crystal display (LCD) in a flat screen televisioncomprises a certain resolution (i.e., the number of dots ofcolor—pixels—on the display) referenced by a horizontal axis (rows) andvertical axis (columns). A wide aspect 22-inch LCD monitor may conformto the WUXGA (Wide Ultra Extended Graphics Array) standard and comprisea resolution of 1920×1200 whereas a smaller device such as a 15-inch LCDmonitor may only conform to the XGA (Extended Graphics Array) standardand comprise a resolution of 1024×768. While an LCD display device isreferenced, the present invention is equally as applicable to aCRT-based display.

Additionally, the particular display mode of a video display device 300determines how many colors that device 300 can display. For example, adisplay that is configured to operate in SuperVGA mode can display16,777,216 colors (also known as true color) the result of a 24-bit-longdescription of a pixel (the bit depth). For example, in 24-bit bitdepth, eight bits are dedicated to each of the three additive primarycolors (RGB). The bit depth therefore determines the number of colorsthat can be displayed at one time. To create a single colored pixel anLCD, for example, uses three sub-pixels with red, green, ad blue filers.Subject to the control and variation of the voltage applied, theintensity of each sub-pixel can range over 256 shades. Combining thesub-pixels produces a possible palette of 16.8 million colors (256shades of red×256 shades of green×256 shades of blue).

The brightness of the light emitted into the user environment can alsobe affected by the size of the outer frame area 320 relative the innerframe area 310. Returning to FIG. 3A, by filling the outer frame area320 with white image data, a certain degree of luminance may be created.Identical white image data (i.e., the same intensity and shade) in FIG.3B, however, will create a greater degree of luminance in that the outerframe area 320 of FIG. 3B is larger than that of FIG. 3A. Thus, theamount of light that is generated by the video display device 300 thougha ‘halo’ may be dynamically controlled during the use of image capturedevice 110.

For example, the image capture device 110 and related client device 130may be in a room with a window in the late afternoon. While there may besufficient natural light to support image data capture at the time gameplay commences, as the afternoon continues and the natural light sourcebegins to disappear (i.e., the sun sets), there may be insufficientlight for the image capture device 110 to properly continue to captureimage data. The outer frame area 320 may, at that point, be illuminatedas a white halo to provide the additional light necessary to allow forgame play to continue. The intensity of the light, as noted, may becontrolled by the client computing device 130 that may be running gamesoftware that has been calibrated with the image capture device 110 todetermine when the proper amount of light is present to allow for thegame to function as intended. In that regard, requisite light settingsmay be predetermined or calibrated based on the particular game and/oruser environment.

As natural light continues to disappear throughout the course of theday, the intensity of the artificial light source emitted from thedisplay device 300 may be gradually increased. For example, a dull graylight source in the outer frame area 320 may provide sufficient light at4.00 PM but a flat white emission may be required from the outer framearea 320 at 6.00 PM. The intensity of the light may also be increased(e.g., the brightness of the light). At some point, the limitations ofthe video display device 300 may be such that even the most intensewhite available does not provide sufficient ‘artificial light’ to allowgame play to continue. At that point, the GPU 210 may cause for the sizeof the inner frame area 310 to decrease and the outer frame area 320 toincrease thereby providing additional artificial light (i.e., additionalindividual pixels in the screen display utilized for illuminationpurposes versus the display of game data). The dynamic adjustmentsdescribed above and in the context of maintaining proper lightingconditions in a user environment are equally applicable in the contextof bandwidth management.

In some instances, particular portions of the user environment may besubject to insufficient lighting conditions. For example, the right sideof a room may have an open window whereby sufficient light may beprovided to allow for image data capture. The left side of the room,however, may have window shades drawn such that no natural light may beentering that portion of the user environment. The orientation of theinner frame area 310 and outer frame 320 may be adjusted such that theinner frame area 310 may be moved to the far right hand side of thedisplay 300 whereby the inner frame area 310 may be off-center butproviding a larger, focused outer frame area 320 on the left-side of thedisplay such that additional artificial light may be generated by thedisplay 300. The positioning and centering of the frame areas 310 and320 may be dynamically adjusted as user of the image capture device 110continues.

Alternatively, particular gradations of color from one-side of thedisplay 300 to another may be implemented wherein the orientation of theinner frame area 310 and the outer frame area 320 may remain constant.In the aforementioned example where the left side of a room is darkerthan the right, the portion of the outer frame area 320 relative theleft side of the room may emit a white light while the right side of theouter frame area 320 displays a neutral background color in thatsufficient natural light is present on the right side of the room. Agradual gradation of color (from white to black) between the left sideof the outer frame 320 to the right side may bridge the two ‘sides’ ofthe outer frame area 320.

Particular portions of the outer frame area 320 (or other frame areas inthe case of multiple frames surrounding the inner frame area 310) may besubdivided into various sub-sections. Each of those sub-sections may besubject to various lighting controls. For example, the outer frame area320 may be sub-divided into four quadrants: an upper right, upper left,lower right, and lower left quadrant. Each of those individual quadrantsmay be subject to various lighting controls (e.g., the upper right andlower left corner of the frame may be illuminated while the upper leftand lower right are not). These various controls include color,luminance, and any other condition that may be controlled by the varioussystems and methods disclosed herein.

The various sub-divisions may also be subjected to various shapes forboth functional and aesthetic reasons. For example, instead of squaresand rectangles, various triangle or other polygonal configurations maybe implemented. Various circular or elliptical patterns may also beused. These and any other shape or design capable of being displayed(e.g., lines, squiggles, waves, dots, splotches, etc.) may be uniformlyor randomly displayed as a group or with a variety of shapes and eachobject subject to lighting control. Various shapes and patterns may alsobe subject to strobing (i.e., regular, controllable series of high powerflashes rather than continuous light) for a variety of effects.

For example, strobing may be used to create a strobe light feature inthe user environment or, similarly, to counter an effect caused bynatural or other artificial light conditions in the user environment.Certain video games or television programs that may cause epilepticfits, feelings of vertigo, and the like and that might be displayed inthe inner frame area 310 may be countered through a counter-strobeeffect in the outer frame area 320. In that regard, the presentlydescribed lighting controls systems and methods may also be implementedwith traditional television programming in addition to ‘chat’ sessionsand video games.

As noted, certain calibrations may take place through the image capturedevice 110. The image capture device 110 may, at start-up, sample theuser environment to determine various lighting conditions and the effectthose conditions will have on image capture over a particular operatingrange (e.g., lowest possible range of capture to an optimal range ofcapture). The image capture device, in conjunction with the variouslighting controls discussed herein as implemented by various hardwareand/or software operations, may then adjust certain user environmentlighting conditions through inner frame area 310 and outer frame area320 lighting manipulations (e.g., size, brightness, orientation, color).During the course of these adjustments, the image capture device 110 may‘re-sample’ the environment to determine what environmental controlcondition will allow for a particular operation of a game or chatsession in light of the particular environment.

The computing device 130 in conjunction with particular ‘chat’ or videogame software may make that determination or such a determination maycome as a result of a management server determination as is described inthe context of FIG. 6. Similarly, the determination of a particular userenvironment condition may come from a user in another environment ifthat particular user is unable to properly view the first user in a chatsession. That decision by a second user may be made by selecting anavailable environment setting in light of several environmental controlconditions.

For example, during the start-up calibration, the first user environmentmay be subject to three different intensities of a particular color oflight; three different colors of light; and three different frame areasizes. The image capture device 110 may capture an image of each ofthose nine possible environments. The effect of that particularenvironment control may then be displayed to the second user in the formof a still frame or even a short (e.g., 5 second) video clip. The seconduser may then select the particular environment control based on theimage or clip that appears best. A user operating in a solo environment(e.g., playing a video game) may select a particular environmentalcontrol in a similar fashion.

These calibrations or adjustments may occur at start-up of a game orchat session or may occur dynamically during the session. For example,adjustments may occur automatically during game play or at breaks suchas between levels. Adjustments may further occur in response to a useraction wherein a query may be made such as during a pause. In someembodiments, a particular frame of data (e.g., an I-frame) may indicatethe propriety and need to readjust lighting conditions in the userenvironment.

Similarly, particular frames of data may be recognized during thedecoding process as likely to cause certain changes in the userenvironment. For example, incoming image data of a conference remoteparticipant may consist of a bright white background that will spillinto the user environment thereby causing an unintended change in userenvironment lighting conditions. These environment changing conditionsmay be identified in advance such that when the actual image data may berendered that display 300 will have already adjusted an inner frame area310 and/or outer frame area 320 as may be appropriate such that noadverse affect or change to the user environment occurs.

It should be noted that in some embodiments of the present invention,the outer frame area 320 may always be present but simply unutilized(i.e., a neutral background color such as black). The frame area maythen ‘filled’ as needed in response to GPU 210 executing a drawinginstruction from the CPU 206. In other embodiments, the actual graphicimages to be displayed may occupy the majority of the game screen and,as such, only the inner frame area 310 may be displayed. If and when anartificial light emission is necessary from a ‘halo’ generated by theouter frame area 320, the software executed by the CPU 206 and otherwisecontrolling the display of the frame areas 310 and 320 and interactingwith the image capture device 110 may cause the GPU 210 to reconfigurethe game environment (e.g., re-size the game environment to fit in asmall inner frame area 310) such that the outer frame area 320 may nowbe displayed and emit the appropriate ‘halo’ of artificial light.

The re-sizing of the various frame areas may occur during the course ofgame play (e.g., the re-sizing occurs as game play progresses).Alternatively, the re-sizing may occur only during pauses in game actionsuch as to avoid the size of the environment and various characterschanging while game play is occurring (i.e., to avoid a vertigo-typeeffect). Additionally, re-sizing may actually cause an automatic pausein game play with a query issued to the user as to whether re-sizing ispermitted. In these instances, the amount of natural light may besufficient to allow game play to continue but not without someprocessing errors. If game play deteriorates to the point thatadditional light is necessary, the user may be asked whether they wishto activate the illumination control feature of the present invention.If not (e.g., the user does not wish to minimize the size of the actualgame environment in the inner frame area 310), the user may manuallyprovide another source of artificial light through, for example, anoverhead lamp or other light source in the actual user environment.Various user input may be provided through, for example, controller 140in addition to other inputs such as voice-recognition.

In some embodiments, different colors or shades of colors may also beutilized to maximize the color balance of a particular user environment.For example, a user may have different light sources offering differentcolors of light. These varying colors may cause difficulty with theimage capture device 110 processing environment information, especiallywith regard to color differentiation. In these instances, the imagecapture device 110 may identify irregularities or processingdifficulties in the environment in a manner similar to theidentification of a lack of sufficient light. The outer frame area 320may used in a similar manner as to produce different colors andintensity of light to counterbalance certain overexposures in theparticular environment. In some embodiments, additional frame areas maybe used (e.g., a third, fourth, or fifth frame area) such thatcombinations of certain colors (e.g., yellow and blue individual halosto create an overall green appearance) may be implemented.

FIGS. 4A and 4B illustrate an exemplary video display device 400reflecting different degrees of illumination intensity in the outerframe area 420 in accordance with one embodiment of the presentinvention. FIGS. 4A and 4B, which reflect a ‘video chat’ session,illustrates the lighting adjustments of a user environment described inthe context of FIGS. 3A and 3B. In that regard, the inner frame area 410of FIG. 4A reflects image data related to another participant (i.e., aremote chat participant); the outer frame area 420 is presently in aneutral background state.

In FIG. 4B, the outer frame area 420 has been ‘filled’ such that it nowreflects white light ‘halo’ as referenced in the context of FIGS. 3A and3B. The inner frame area 410 has also been reduced in that the lightingconditions in the environment of the user receiving the present imagehave made it necessary to increase the amount of light emitted into thatenvironment. As such, the outer frame area 420 was expanded at theexpense of the inner frame area 410.

In some embodiments of the present invention, the inner frame area 410may need to be reduced in that the graphic data in the inner frame area410 is inadvertently creating excess light and thereby over saturatingthe user environment. For example, in the aforementioned baseball gameexample, if the user hits a home run, the image displayed may pan upwardto the sky to follow the path of the baseball as it travels toward theoutskirts of the ballpark. As the scene pans toward the sky, the screendisplay will become predominantly white and gray (i.e., colors of thesky and clouds). These bright colors may ‘spill’ into the userenvironment, which may cause image data capture difficulties. Inresponse to such a situation, the inner frame area 410 may decrease insize to reduce the amount of secondary light ‘spillage’ into the userenvironment. Additionally, the outer frame area 420 may be darkened tohelp counter suddenly excess bright light being emitted from the innerframe area 410.

FIG. 5A illustrates an exemplary method 500 for implementing lightingcontrol of a user environment through varying illumination intensity ina video display device in accordance with one embodiment of the presentinvention.

In step 510 an image capture device may attempt to capture image datafrom a user environment. In step 520, a determination may be madewhether adverse lighting conditions are inhibiting the processing of theimage capture data or the actual capture of the data. If there are noadverse conditions present and the image data is captured and processedwithout incident, image data capture proceeds as necessary followed bysubsequent processing of the data.

If it is determined in step 520 that adverse lighting conditions doexist, an attempt may be made to adjust display intensity in step 530.This may occur through various adjustments whereby the outer frame areamay be ‘filled’ with a particular color and intensity of light asdiscussed in the context of FIGS. 3A and 3B and 4A and 4B above. If theadjustment as to display intensity is not sufficient, then (in step540), frame image boundaries may be enlarged (or reduced as isappropriate) in order to increase or decrease the total amount ofartificial light projected into the user environment.

While step 540 and the adjustment of frame boundaries is recited asfollowing step 530 and the adjustment of light intensity, this is not tosuggest the necessity of a step-by-step process. Both adjustments mayoccur concurrently or frame adjustment may occur prior to intensityadjustment in that FIG. 5A is an exemplary embodiment only. Regardlessof the order, following the adjustment of image intensity and/or frameboundaries, further attempts at image capture and/or image dataprocessing occur as the method repeats or continues (as is appropriate)in step 510.

FIG. 5B illustrates an exemplary method 550 for managing bandwidth withrespect to the generation and transmission of image capture data inaccordance with one embodiment of the present invention. Specifically,FIG. 5B illustrates a method 550 for managing bandwidth through theimplementation of lighting control of a user environment through, forexample, varying illumination intensity in a video display device.

As discussed in U.S. patent application Ser. No. 11/624,886, the controlof lighting conditions in a user environment may be associated withbandwidth and processing availability. In MPEG compression, images arerepresented in YUV color space (YCbCr) wherein 24-bits per pixel arepresent: 8-bits for luminance (Y) and 8-bits for each of the twochrominance (U), (V) elements. The chrominance information in the YUVcolor space data may be sub-sampled in both the horizontal and verticaldirection. All of the luminance information may be retained, however, asthe human eye is more sensitive to luminance information rather thanchrominance information. Frames of video are subsequently divided into16×16 macro-blocks consisting of four 8×8 luminance blocks and two 8×8chrominance blocks (1U and 1V).

Each frame of video data may then be encoded as one of three types offrames: intra-frame (I-frames), forward predicted frame (P-frames), andbi-directional predicted frames (B-frames). In the case of an I-frame,the frame may be encoded as a single image with no reference to past orfuture frames. Further, with the exception of data quantizationfollowing the encoding of each 8×8 block from a spatial domain to afrequency domain utilizing the Discrete Cosine Transform (DCT) and theaforementioned sub-sampling, there is no lossy compression in anI-frame—especially with regard to luminosity. The other frametypes—P-frames (relative a past reference frame) and B-frames (relativea past reference frame, a future reference frame or both)—refer to theI-frame. Thus, decreasing certain luminosity information in the I-framewould represent savings with regard to the all of the frames in a streamof video data.

In this regard, the image data captured from the user environment couldbe accepted at the lowest possible operating condition. That is,lighting controls would illuminate a user environment no more so thannecessary in order to reduce the amount of luminosity data beingcaptured, compressed and subsequently transmitted over a network. Thisdecreased luminosity data represents not only savings in bandwidth at atransmitting computing device, a receiving computing device, and overthe network backbone but can also recognize a savings with regard toprocessing cycles at a computing device perform digitization andcompression following image capture.

Similarly, certain colors captured from a particular user environmentmay be captured and/or cancelled out by adjusting display emissionsand/or frame boundaries as discussed above. For example, in the contextof a YUV 4:4:4 environment—where Y pertains to luminance (brightness)and U and V are chrominance (color) components—each channel is expressedwith 8 bits (256 possible values) and uses the same amount of space. Asnoted above, the human eye is most sensitive to luminance—the black andwhite information in picture detail. Less sensitive areas of the humaneye “fill in” color. That is, color information may be reduced withoutthe human eye noticing the same. Thus, the chrominance components (U)and (V) may be reduced without the human eye adversely noticing thesame. Bandwidth consumption may be reduced by adjusting chrominanceinformation.

For example, the environment may be effectively captured at (ormanipulated to conform to) the equivalent of 4:2:2 format whereby thechroma components are sampled at half the rate of luma thereby reducingbandwidth of the signal by as much as one-third with little noperceptible visual difference. Specifically, red and blue channels ofthe video signal may be half the bandwidth of the luminance information.For example, green may be cancelled out as green can be calculated fromred and blue chrominance information in the context of luminanceinformation.

A utility related to preservation of bandwidth and processing powervis-à-vis the image capture device and/or related hardware or softwaremay be manually implemented by a user. For example, a user may recognizethat their particular computing device lacks accelerated processingpower or may be coupled to a low bandwidth network or, similarly, that arecipient of data captured by the image capture device suffers fromsimilar limitations. In these instances, the user may activate such autility to avoid bottlenecks of data.

Similarly, this preservation utility may be implemented on an as-neededbasis, the need being recognized by a network congestion monitoringutility like ping, which utilizes the Internet Control Message Protocol(ICMP), and trace route, which utilizes the Uniform Datagram Protocol(UDP), to measure network response time. These and other networkutilities may be implemented in various network devices (e.g., switchesor routers) or through large-scale platforms like the NetworkApplication Performance Analysis Solution (NAPAS) for Cisco Systems.Indicia of network congestion generated by these applications andsolutions may be provided to a computing device operating certainsoftware related to image capture (e.g., a client computing device or aconference management server) such that bandwidth and/or processorpreservation may be activated through the manipulation of lightingconditions that are not optimal but otherwise sufficient to allow forcontinued image data capture.

In certain embodiments wherein one end of a video data exchange degradesa particular lighting condition for the purposes of improved processingor transmission time, the recipient of that data may adjust the incomingvideo data such that it does not appear as lesser quality image data.For example, the incoming video data may be pre-processed such thatluminance that was degraded by the sender of the data may beartificially amplified by the recipient through editing or videopre-processing editing software that automatically adjusts the data asit is decoded and rendered.

In step 560, the bandwidth being consumed by image capture datatransmitted over a network (as may occur in the context of a video chatsession like that described in FIG. 6 below) is made. A determination ofbandwidth may be made utilizing, for example, the aforementioned NAPASor simpler methodologies such as ping and trace route. The determinationmay occur at the transmitting computing device, at the receivingcomputing device, or at some intermediate computing device or at someother point in a communications network. The information may then beprovided to the transmitting device such that it may adjust its imagecapture operations as is appropriate and is discussed in the context ofsteps 580 and 590. Alternatively, another device may make certaindetermination as to bandwidth consumption and provide instructions tothe transmitting computing device such that it may responsively adjustits image capture operations as may occur through screen and/or frameadjustment.

In step 570, a determination is made as to whether image capture datathat is being transmitted over a network exceeds a particular bandwidthlimitation. The particular bandwidth may be a static predeterminedamount (e.g., a target threshold). That static threshold may be manuallyset by a user or network administrator. The static threshold may also beidentified at the outset of a chat session and remain constantthroughout the session. Alternatively, the amount may be determined inthe context of a particular video chat session and may change during thecourse of the chat session. That is, as network conditions change fromthe beginning to the end of the session and at various points inbetween, the target threshold may be adjusted subject to those changingnetwork conditions.

In one example, two users chatting over a dedicated corporate networkfor such communications may be unlikely to encounter other users orapplications attempting to utilize the same finite amount of bandwidthin the network. As such, even if captured image data consumes largeamounts of bandwidth, the availability of large amounts of networkoverhead may obviate the need to otherwise adjust network trafficconsumption. On the other hand, if two users are chatting over a publicnetwork shared by other users at a peak traffic time, even image capturedata that is ‘normal’ under the circumstances may exceed a particularbandwidth allotment or may otherwise encounter transmission difficultiesdue to other users and applications sharing the network. Further, if achat session commences amongst a group of fifteen participants,bandwidth consumption may initially be high with respect to the groupand available resources. As participants drop out of the conference(e.g., their presence is no longer required), the total bandwidthconsumption may decease and what constitutes an excessive use ofbandwidth for any one user may change as additional resources becomeavailable to the session.

If a determination is made that image capture data does not exceed aparticular bandwidth (either predetermined or determined in a dynamicfashion), then measurement continues in step 560. Measurement ofbandwidth consumption may occur at predetermined times during the chatsession (e.g., every five minutes). Measurement may also occur upon theoccurrence of a particular event, for example, transmission of theI-frame or every n^(th) I-frame. When and how often measurement occursmay also be set by the user or a network administrator.

If it is determined that image capture data being transmitted over thenetwork exceeds a particular bandwidth limitation, adjustment of displayintensity and frame boundaries may occur in steps 580 and 590, Theseparticular adjustments may be similar with respect to the display andframe adjustments that occurred in steps 530 and 540 in the context ofFIG. 5A. Like FIG. 5A, adjustment of display intensity and frameboundaries are not necessarily a step-by-step process. Both adjustmentsmay occur concurrently or frame adjustment may occur prior to intensityadjustment. Further, only one particular adjustment (i.e., frame but notdisplay) may occur.

Lighting adjustment sources other than or in addition to the screen anda frame area within the screen may be utilized in the context ofmanaging bandwidth consumed by image capture data. For example, anexternal light source may be coupled to a device controlling lightingemissions (e.g., client computing device 130 of FIG. 1). This couplingmay occur in the context of a USB connection or other data connection.The external lighting device may be proximate or otherwise remote fromthe actual image capture device and related computing device so long asthe light source may be configured to result in some influence of thelighting conditions of a particular user environment. Some lightingcontrol hardware may be integrated with particular display device suchas a backlight or other ambient light projector.

FIG. 6 illustrates an exemplary network 600 of real-time,three-dimensional, interactive environment data generating systems610/620 as may be utilized in a ‘video chat’ session or interactivevideo game and in accordance with one embodiment of the presentinvention. Systems 610/620 exchange data over communications network630. Optional conferencing management server 640 may facilitateexchanges between systems 610/620 during the course of a chat or gameplay session.

Systems 610/620 may comprise an exemplary system like that disclosed inFIG. 1. In that regard, such a system may comprise an image capturedevice, a video display device, and a client computing device havingprocessor functionality. Systems 610/620 would be placed in the contextof a user environment, such as a living room or a conference room, whichwould be located within the field-of-view of the image capture device.

Systems 610/620 may each be running software for facilitating avideo-conference or video ‘chat’ session or otherwise related to aparticular video game involving image data capture. The exchange of databetween systems 610/620 may be limited to video images and accompanyingaudio. The session may further comprise additional ancillary data suchas textual exchanges (e.g., closed captioning); various in-conferenceapplications such as an enterprise application allowing forcollaborative development between multiple users (e.g., a wordprocessing document or presentation slide); video-games that may beplayed on-line during the session; or other multi-media applications(e.g., a media player). These data exchanges are in addition to varioussession protocols related to establishing and maintaining the exchangeof conferencing and other related data.

Through the aforementioned software, which may be accessed throughoptical disk control unit 228, the network adaptor 240 may initiate aconferencing session through, for example, Ethernet connection 242 overthe aforementioned network 630. Image data (and related audio data) maybe captured through the aforementioned image capture device, which maybe coupled to the system 610 via, for example, USB interface 220. GPU210, in turn, executes various drawing instructions related to not onlythe actual conferencing data (e.g., user images) but also the innerframe area and outer frame area and related illumination controlcommands (e.g., ‘filling’ the outer frame with a particular intensityand color) whereby lighting conditions of a particular environment maybe controlled. Instructions related to control of lighting conditions inthe particular environment may be a part of the video conferencingsoftware executed by each system 610/620 or at server 640.

Systems 610/620 may also be configured to manage bandwidth consumptionbased on certain bandwidth measurements taken by each system 610/620.Bandwidth measurements may also be taken by a third-party application orremote computing device (e.g., server 640) and reported to each system610/620. Systems 610/620 may adjust certain lighting conditions throughscreen and frame adjustments in response to locally determinations madewith respect to network conditions and bandwidth consumption.Alternatively, systems 610/620 may be configured to receive instructionsfrom a remote computing device such as server 640 or another conferenceparticipant's computing device and adjust certain user environmentlighting conditions in response to the same.

In some embodiments of the present invention, address book data may bestored on a removable data storage device as may be accessed throughmemory card interface 218. This ‘address book’ data may be related toother users of the present video-conferencing/chat system 600. Theconferencing software may also include certain security features orparental controls to ensure that under-age children do not access thesystem 600 or are limited to sessions with particular, pre-approvedconferees. Such information may also be stored on the aforementionedmemory device. Conferencing software may further facilitate thegeneration of video mail such as a 30-second full-motion video. Softwaremay further facilitate entry into various chat-rooms or interaction withcertain applications, such as video games.

Communications network 630 comprises various communications facilitiesand mediums (e.g., telephony, wireless, satellite, cable, optic, and soforth) as may be provided by telecommunications companies and InternetService Providers. Communications network 630 may be a geographicallywidespread network (e.g., a Wide Area Network (WAN)), like the Internetthat depends upon the aforementioned communications facilities to linkvarious network segments. In that regard, a WAN may ultimately becomprised of smaller linked communications networks such as Local AreaNetworks (LANs). A LAN is typically comprised of a group of computersand other devices dispersed over a relatively limited area and connectedby, for example, a variety of broadband communications links. LANs maytake on a variety of configurations including server client,peer-to-peer, peer groups, or a combination of the same. Communicationsnetwork 630 may also be a closed, proprietary network.

In some embodiments of the present invention, a conferencing managementserver 640 may be utilized to conduct various video conferences or otheruser interactions. For example, system 610 may actually connect withserver 640; server 640 will then create a connection with system 620 anda ‘chat’ session may commence. Server 640 may provide various networkcontrols such as connection acceptance, session initiation andtermination, bandwidth management, conference recording (e.g., recordingthe conference session or a transcript of the same for future access);managing chat rooms or group sessions (e.g., sessions involving morethan two participants); facilitating certain data exchanges ancillary tothe actual conference (e.g., documents, presentations, video games); andalso ensuring proper control of lighting conditions in a particular userenvironment.

In the latter example, the server 640 may monitor lighting conditions inthe conferencing environment of each system (610/620) and adjust certainframe settings through the necessary drawing instructions such thatproper lighting conditions are maintained at both ends of theconference. In this regard, server 640 may actually provide the drawinginstructions for execution by the appropriate system to ‘fill’ a frameor adjust the size of the same. Server 640 may also receive variousindications from each system (610/620) concerning the quality oflighting conditions at the other system (610/620) and facilitateadjustments as is appropriate by communicating the various indicationsto various participant devices.

Server 640 may also manage bandwidth consumption in a similar fashionthrough the control of lighting conditions in various user environments.Server 640 may include a bandwidth measurement application for measuringsuch conditions and responding in an according fashion (i.e., adjustinglighting conditions in a user environment through screen and/or frameadjustments). Server 640 may, alternatively, receive information fromeach system (610/620) or a third-party application in the communicationsnetwork 630 concerning various network conditions and bandwidthconsumption of the same. Measure of bandwidth conditions may occur overa network 630, at the origin and/or destination of video conference/chatinformation, at the server 630, as well as at and between various hopsin network 630. Server 640 may adjust lighting conditions to allow forbandwidth consumption control in response to a measurement at one ormore points in the network 630 as may be deemed appropriate in responseto certain measurement information.

Server 640 may also host the necessary software for the conference totake place. In such an instance, the systems 610/620 may comprise onlyminimal application data necessary to contact the server 640 to indicatethe desire to participate in a conference session. Critical conferencingarchitecture software and execution of the same may then occur at theserver 640, which may have considerably more processing power than theend-user device (e.g., an entertainment system). By leaving heavyprocessing to the more powerful conferencing server (like server 640),conferencing exchanges do not ‘slow down’ or experience jitter due toprocessing delays at the various end-users and their related computingdevices. In some embodiments of the present invention, like thoseutilizing a high speed processing model as disclosed in U.S. patentpublication number 2002-0138637 for a “Computer Architecture andSoftware Cells for Broadband Networks,” server 640 may also managevarious distributed computing operations.

While the present invention has been described with reference toexemplary embodiments, it will be understood by those skilled in the artthat various changes may be made and equivalents may be substituted forelements thereof without departing from the true spirit and scope of thepresent invention. In addition, modifications may be made withoutdeparting from the essential teachings of the present invention. Variousalternative systems may be utilized to implement the variousmethodologies described herein and various methods may be used toachieve certain results from the aforementioned systems. Certainmethodologies and benefits discussed in the context of lighting controlof a user environment for detection of user interactions may be equallyas applicable to controlling bandwidth consumption with respect tocaptured and transmitted image data and vice versa.

The steps of the processes disclosed herein and various alternatives ofthe same may be embodied in hardware or software. Such embodiments mayinclude a computer-readable medium such as an optical disc, memory card,carrier waves, and the like. Such mediums may include instructionsexecutable by the processor of a computing device.

1. A method for implementing lighting control of a user environment themethod comprising: capturing image data from a user environment via animage capture device, wherein the image data includes data about areal-world activity performed by a user interacting with a gameenvironment, and wherein the image capture device is associated with adisplay device for implementing lighting control of the userenvironment; executing instructions stored in memory wherein executionof instructions by a processor: determines that an adverse lightingcondition exists, the adverse lighting condition preventing proper datacapture by the image capture device; and adjusts a lighting intensity ofthe display device for proper data capture in response to the adverselighting condition, wherein adjusting the lighting intensity includesincreasing or decreasing a total amount of light projected into the userenvironment by the display device.
 2. The method of claim 1, whereinadjusting the lighting intensity of the display device includesadjusting the size of an inner frame area or outer frame area, thedisplay device having an inner frame area and an outer frame area foruse in implementing lighting control of the user environment.
 3. Themethod of claim 1, wherein adjusting the lighting intensity of thedisplay device includes adjusting the orientation of the inner framearea or outer frame area of the display device.
 4. The method of claim1, wherein the user performs the real-world activity using a propdevice.
 5. The method of claim 1, wherein adjusting the lightingintensity of the display device occurs during game play.
 6. The methodof claim 5, wherein adjusting the lighting intensity of the displaydevice occurs following a user approval of the intended adjustment. 7.The method of claim 1, wherein adjusting the lighting intensity of thedisplay device further includes adding an additional frame area.
 8. Themethod of claim 1, wherein adjusting the intensity of the display deviceincludes adjusting the color of the projected light to maximize thecolor balance of the user environment.
 9. The method of claim 8, whereinadjusting the color of the projected light includes muting, countering,or cancelling the color.
 10. The method of claim 1, further comprisingcalibrating the lighting intensity of the user environment to determinea lighting control condition in the user environment.
 11. A system forimplementing lighting control of a user environment, comprising: animage capture device for capturing image data from a user environment,wherein the image data includes data about a real-world activityperformed by a user interacting with a game environment; a computingdevice that: determines that an adverse lighting condition exists, theadverse lighting condition preventing proper data capture by the imagecapture device, and adjusts a lighting intensity of the display devicefor proper data capture in response to the adverse lighting condition,wherein adjusting the lighting intensity includes increasing ordecreasing a total amount of light projected into the user environmentby the display device; and a display device that displays image datareceived from the computing device and project light into the userenvironment.
 12. The system of claim 11, wherein the display deviceincludes an inner frame area and an outer frame area for implementinglighting control of the user environment.
 13. The system of claim 12,wherein the image data is displayed in the inner frame area.
 14. Thesystem of claim 12, wherein a portion of the inner frame area or outerframe area is subdivided into subsections, each subsection subject tolighting control.
 15. The method of claim 11, wherein the image capturedevice is associated with a prop device for interacting with the gameenvironment, the user using the prop device to perform the real-worldactivity.
 16. The system of claim 15, wherein the image capture deviceis configured to track the prop device and translate the real-worldactivity performed by the user into on-screen display data.
 17. Thesystem of claim 11, further comprising a lighting source external to thedisplay device for projecting additional light into the userenvironment.
 18. The system of claim 11, wherein the computing deviceadjusts the lighting intensity of the display device by adjusting thecolor of the projected light to maximize the color balance of the userenvironment.
 19. A non-transitory computer readable storage mediumhaving a program embodied thereon, the program being executable by aprocessor to perform a method for implementing lighting control of auser environment, the method comprising: capturing image data from auser environment, wherein the image data includes data about areal-world activity performed by a user interacting with a gameenvironment; determining that an adverse lighting condition exists, theadverse lighting condition preventing proper data capture; and adjustinga lighting intensity for proper data capture in response to the adverselighting condition, wherein adjusting the lighting intensity includesincreasing or decreasing a total amount of light projected into the userenvironment.